Stereoscopic video providing method and stereoscopic video display

ABSTRACT

A transmitting-side performs an image analysis on a photographed two-dimensional image ( 100 ) and extracts a background image ( 101 ), an image ( 102 ) of a building, and an image ( 103 ) of an automobile. These extracted images are handled as objects. Moreover, a depth map is generated by applying a depth value to each dot. In addition, thickness information is applied. The thickness information may be applied to each dot or to each object.

TECHNICAL FIELD

The present invention relates to a stereoscopic vision-use imageproviding method and a stereoscopic image display device.

PRIOR ART

As a prior art, there is proposed a stereoscopic image receiver and astereoscopic image system-that generate a stereoscopic image on thebasis of depth information extracted from a two-dimensional video signaland the two-dimensional video signal (see Japanese Patent Laying-openNo. 2000-78611).

With the above-described prior art, it is possible to generate astereoscopic vision-use image that is allowed to have parallaxinformation from an actually photographed two-dimensional image. Herein,for example, in a case that a house exists as an object in thetwo-dimensional image and this image is combined with an image in whicha ball is rolling, if the ball comes to a position where the ball hitsthe house from a lateral direction, the ball is to be so displayed as tohit the house and bounce off. In the above-mentioned prior art, asurface position of the object is merely defined by depth information,and it is impossible to determine a collision between the ball and theobject. Accordingly, the ball is so displayed as to pass through infront of the house, or as to pass behind the house.

SUMMARY OF THE INVENTION

In view of the foregoing circumstances, it is an object of the presentinvention to provide a stereoscopic vision-use image providing methodand a stereoscopic image display device that allow various stereoscopicdisplays by adding further information regarding an object, and so on.

In order to solve the above-mentioned problem, the stereoscopicvision-use image providing method of the present invention ischaracterized in providing, when providing the two-dimensional image asdata, stereoscopic vision-use information useful for converting the dataof the two-dimensional image into a stereoscopic vision-use image andthickness information of an object on the two-dimensional image, asadditional information of the two-dimensional image together with thedata of the two-dimensional image.

With the above-described configuration, by thickness information of anobject on a two-dimensional image, it is possible to handle the objectas the object having thickness also on a stereoscopic vision-use image.As a result, it is possible to utilize the information, in a case ofcomposing the stereoscopic vision-use image with an alternate image, forexample, for determining a collision between the object on thestereoscopic vision-use image and the alternate image (or between theobject on the stereoscopic vision-use image and an object on thealternate image), and so on.

Moreover, the stereoscopic vision-use image providing method of thepresent invention is characterized in providing, when providing atwo-dimensional image as data, stereoscopic vision-use informationuseful for converting the data of the two-dimensional image into astereoscopic vision-use image such as depth information indicating anear side position of an object on the two-dimensional image and depthinformation indicating a far side position of the object on thetwo-dimensional image, as additional information of the two-dimensionalimage together with the data of the two-dimensional image.

Also with such the configuration, by depth information indicating a nearside position of an object on a two-dimensional image and depthinformation indicating a far side position of the object on thetwo-dimensional image, it is possible to handle the object as an objecthaving thickness also on the stereoscopic vision-use image.

Moreover, the stereoscopic vision-use image providing method of thepresent invention is characterized in providing, when providing atwo-dimensional image as data, stereoscopic vision-use informationuseful for converting the data of the two-dimensional image into astereoscopic vision-use image and thickness information of each dot onthe two-dimensional image as additional information of thetwo-dimensional image together with the data of the two-dimensionalimage.

With the above-described configuration, by thickness information of eachdot on the two-dimensional image, it is possible to handle each dot as adot having thickness also on the stereoscopic vision-use image. As aresult, it is possible to utilize the information, in a case ofcomposing the stereoscopic vision-use image with an alternate image, forexample, for determining a collision between a displayed object on thestereoscopic vision-use image and a displayed object on the alternateimage, and so on.

Furthermore, the stereoscopic vision-use image providing method of thepresent invention is characterized in providing, when providing atwo-dimensional image as data, stereoscopic vision-use informationuseful for converting the data of the two-dimensional image into astereoscopic vision-use image such as depth information indicating anear side of each dot on the two-dimensional image and depth informationindicating a far side of the each dot on the two-dimensional image, asadditional information of the two-dimensional data together with thedata of the two-dimensional image.

Also with such the configuration, by depth information indicating a nearside position of each dot on the two-dimensional image and depthinformation indicating a far side position of each dot on thetwo-dimensional image, it is possible to handle each dot as the dothaving thickness.

In these stereoscopic vision-use image providing methods, informationmay be provided by any one of methods such as broadcasting, acommunication, and a recording into a recording medium. In addition, atleast one photographing time information out of focal distanceinformation and field angle information may be provided as additionalinformation of the two-dimensional image together with the data of thetwo dimensional image.

Moreover, a stereoscopic vision-use image providing method of thepresent invention is a stereoscopic vision-use image providing methodthat provides multi-viewpoint two-dimensional images as data, and ischaracterized in providing at least one photographing time informationout of information indicating intervals between viewpoints, informationindicating an angle formed of adjoining viewpoints and an object to bephotographed, information indicating a cross location of optical axes,focal distance information, and field angle information as additionalinformation of the two-dimensional image together with the data of thetwo-dimensional image.

With the above-described configuration, it is possible that a displaydevice utilizes the photographing time information provided asadditional information of the two-dimensional image, and the deviceselects a viewpoint depending on a position of an object to bephotographed, for example. In addition, in a case that themulti-viewpoint is obtained by photographing an-object to bephotographed by cameras arranged in a circular shape around the objectto be photographed, it becomes easy to incorporate a stereoscopic imageof the object to be photographed into a three-dimensional data andhandle the stereoscopic image.

Furthermore, a stereoscopic image display device comprises a means forgenerating data of a stereoscopic vision-use image on the basis of dataof a two-dimensional image and stereoscopic vision-use information, ameans for composing an alternate image with the stereoscopic vision-useimage on the basis of data of the alternate image, and a means fordetermining a collision between a displayed object on the stereoscopicvision-use image and a displayed object on the alternate image on thebasis of thickness information of dots or an object on thetwo-dimensional image that are additional information of the twodimensional image.

With the above-described configuration, thickness information of anobject on a two-dimensional image allows the object to be handled as theobject having thickness also on the stereoscopic vision-use image, andin composing with an alternate image, a determination of collision isperformed, so that it is possible to perform a process according to thedetermination of the collision.

Moreover, a stereoscopic image display device of the present inventioncomprises a means for generating data of a stereoscopic vision-use imageon the basis of data of a two-dimensional image and depth informationindicating a near side position of an object on the two-dimensionalimage, and a means for generating thickness information of the object onthe basis of depth information indicating a far side position of theobject and the depth information indicating the near side position ofthe object.

Furthermore, a stereoscopic image display of the present inventioncomprises a means for generating data of a stereoscopic vision-use imageon the basis of data of a two-dimensional image and depth informationindicating a near side position of each dot on the two-dimensionalimage, and a means for generating thickness information on the basis ofdepth information indicating a far side position of the each dot and thedepth information indicating the near side position of the each dot.

In addition, a stereoscopic image display of the present invention is astereoscopic image display that performs a stereoscopic image displayusing two images out of multi-viewpoint images, and is characterized inselecting the two images on the basis of at least one photographing timeinformation out of viewpoint intervals information, informationindicating an angle formed of adjoining viewpoints and an object to bephotographed, information indicating a cross location of optical axes,focal distance information, and field angle information.

With the above-mentioned configuration, it is possible to select the twoimages on the basis of the photographing time information provided asadditional information of the two-dimensional image. For example, in acase that the object to be photographed is in a position close to anobserver, two images are selected so that intervals between viewpointsare large. In a case that the object to be photographed is in a positionfar from the observer, two images are selected so that intervals betweenviewpoints are small.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1(a), (b), and (c) are descriptive diagrams showing a stereoscopicvision-use image providing method of an embodiment of the presentinvention;

FIGS. 2 (a) and (b) are descriptive diagrams illustrating a transmissionformat of a stereoscopic vision-use image;

FIG. 3 is a descriptive diagram showing a collision determination;

FIG. 3(a) shows an image;

FIG. 3(b) shows a case that there is thickness information;

FIG. 3(b) shows a case that there is no thickness information;

Each of FIGS. 4(a), (b) and (c) is a descriptive diagram showing anobtainment of multi-viewpoint images (multi-eye images); and

FIG. 5 is a descriptive diagram showing a selective form of two images.

BEST MODE FOR PRACTICING THE INVENTION

Hereinafter, a stereoscopic vision-use image providing method and astereoscopic image display device will be described referring to FIGS. 1to 5.

At first, on the basis of FIG. 1, a generation of a stereoscopic imageby a two-dimensional image and a stereoscopic vision-use information(here, depth information is used as the stereoscopic vision-use image),and a determination of a collision based on thickness information of anobject on the two-dimensional image and a composite image will bedescribed. It is noted that, in this Figure, a system will be describedas the system that is formed of a transmitting-side structured as abroadcasting station, a server on the Internet, or the like, and areceiving-side formed of a broadcasting receiver, a personal computerprovided with the Internet access environment, or the like.

FIG. 1(a) shows an actually photographed two-dimensional image 100. Onthe transmitting-side, an image analysis is performed on thetwo-dimensional image 100, and as shown in FIG. 1(b), a background image101, an image 102 of a building, and an image 103 of an automobile areextracted. These extracted images are handled as objects (for example,edge information). Moreover, a depth value is applied for each dot andthe depth map is generated. It is noted that it is also possible toapply the depth value to each object. The depth value may be appliedautomatically (presumptively), or may be applied by a manual procedure.

Furthermore, the thickness information is applied. The thicknessinformation may be applied to each dot or each object. In a case that athickness of an object is almost fixed (for example, in a case that anobject is a box-shaped building photographed from a front side, and thelike), it is permissible to apply the thickness information to eachobject. Moreover, two depth maps may be applied. As a result of onedepth map being depth information indicating a near side position andthe other depth-map being depth information indicating a far sideposition, the thickness is found by the difference between the near sideposition and the far side position. In addition, in the case that twodepth maps are applied, the depth information may be changed over suchthat the depth information indicating the near side position and thedepth information indicating the far side position change alternately,for example, in a case of the two dimensional image of a moving picture,depth information indicating the near side position is applied to thetwo-dimensional image of a certain frame and depth informationindicating the far side position is applied to the two dimensional imageof the next frame.

Thus, the transmitting-side, when providing the two-dimensional image asdata, transmits the depth map and the thickness information asadditional information of the two-dimensional image together with thetwo-dimensional image data. In transmission, a process for compressingthe data and a process for multiplexing are performed. One example of aformat for inserting the thickness information is shown in FIG. 2 (a).In this format, a property of the information is indicated in an“identification part”, and in this case, the information indicates thedepth information and the thickness information. A “dot number”specifies each dot. The “depth information” is a depth value of a dotindicated by the dot number. The “thickness information” is thethickness information of a dot of the dot number.

Or, the transmitting-side, when providing the two-dimensional image asdata, provides the depth map indicating the near side position and thedepth map indicating the far side position as additional information ofthe two-dimensional image together with the two-dimensional image data.One example of a format in this case is shown in FIG. 2(b). In thisformat, the property of information is indicated in the “identificationpart”, and in this case, the information indicates the depthinformation. The “dot number” specifies each dot. “First depthinformation” is a depth value of the near side of the dot indicated bythe dot number. “Second depth information” is a depth value of the farside of the dot indicated by the dot number.

As shown in FIG. 1 (c), the receiving-side receives each of the dataincluding the background image 101, the image 102 of the building, andthe image 103 of the automobile, and the additional information. Ifthese data are multiplexed, a demultiplexing process is performed. As adecoding process toward each data, basically, the process based on MPEG4, or the like, is adopted, for example. In addition, the receiving-sidegenerates images 104R for a right eye and 104L for a left eye to which aparallax is applied based on each of the data including the backgroundimage 101, the image 102 of the building, and the image 103 of theautomobile, the depth map and a composition-use image (for example, athree-dimensional image of a ball 105 generated by a computer).Accordingly, the receiving-side is provided with a means (a modem, atuner, etc.) for receiving data, a demultiplexer, a decoder, astereoscopic image data generating part for generating the stereoscopicvision-use image data based on the two-dimensional image data and thestereoscopic vision-use information, and an image composition processingpart for composing an alternate image with the stereoscopic vision-useimage based on data of the alternate image. In addition, in thisembodiment, the receiving-side is provided with a collision determiningpart for determining a collision between a displayed object on thestereoscopic vision-use image and a displayed object on the alternateimage.

In the collision determining part, the following process is performed.Here, in order to simplify the description, as shown in FIG. 3(a), it isassumed that the depth value of the background image 101 is 100, thedepth value and the thickness value of the image 102 of the building are50 and 30, respectively, the depth value and the thickness value of theimage 103 of the automobile are 30 and 10, respectively, the depth valueand the thickness value of the ball 105 as the composition-use image are55 and 1, respectively. On the basis of such the information, as shownin FIG. 3(b), it is possible to judge that the ball 105 is located on acoordinate at a rear side of the image 103 of the automobile and on acoordinate between a surface side and a rear side of the image 102 ofthe building. Furthermore, the case that only the conventional depthvalue is applied is shown in FIG. 3(c) for reference. As understood fromthese Figures, with an embodiment of the present invention, when dotsthat form a moving end of the rolling ball 105 are located on the dotsthat form a side surface of the image 102 of the building, it isdetermined that the ball 105 collided against the image 102 of thebuilding. This determination result is applied to the aforementionedcomputer, and the computer generates a three-dimensional image of theball 105 in which a moving course of the ball 105 is reversed (bouncedoff). It is noted that, in the case that only the depth value isapplied, an image in which the ball 105 passes at the rear side of theimage 102 of the building is generated.

Next, an obtainment of multi-viewpoint images (multi-eye images) will bedescribed. FIG. 4(a) shows a state at the time of obtaining themulti-viewpoint images (actually photographed). In this Figure, anobject A to be photographed (object) is photographed by a camera 1, acamera 2, a camera 3, a camera 4, a camera 5, and a camera 6, so that itis possible to obtain the two-dimensional image with six viewpoints.Then, in transmitting this two-dimensional image with six viewpoints asdata, at least one photographing time information out of informationindicating intervals between viewpoints (intervals of cameras),information indicating a cross location of optical axes, focal distanceinformation (distance to an object), and field angle information aretransmitted as additional information of the two-dimensional imagetogether with the two-dimensional image data. FIG. 4(b) shows anotherexample of the time of obtainment of the multi-viewpoint images(actually photographed). In this example, by arranging a camera 11, acamera 12, a camera 13, a camera 14, a camera 15, a camera 16, a camera17 and a camera 18 circularly around the object A, and photographing theobject, the multiple-viewpoint two-dimensional image is obtained. Inthis case, instead of the information indicating the intervals betweenthe viewpoints, information indicating an angle formed between theadjoining viewpoints (cameras) and the object A to be photographed isobtained. Moreover, as shown in FIG. 4(c), also by photographing theobject A to be photographed using one camera while rotating the object,it is possible to obtain the multi-viewpoint two-dimensional image. Atthis time, a rotation speed may be included in the photographing timeinformation. As a result of the photographing time information beingapplied together with the multiple-viewpoint two dimensional imageobtained by the methods shown in FIGS. 4(b), 4(c), it is possible toapply a three-dimensional coordinate value to each point (each dot ofdisplayed image) that form a surface of the object A to be photographed.Accordingly, it becomes easy to incorporate the object A to bephotographed (actually photographed) into three-dimensional data andhandle the object (it becomes easy to arrange an actually photographedimage in the three-dimensional data). In this case, it is preferable torender a background black (a black curtain is arranged on thebackground) and to photograph so as to take out one object.

A stereoscopic image display device to which the multi-viewpoint twodimensional data and the photographing time information are appliedperforms a stereoscopic image display by using two images out of themulti-viewpoint images. As the stereoscopic image display method usingtwo images, there are such methods as to display two images alternatelyin terms of time and see the images with shutter eyeglasses, display twoimages alternately in terms of space and see the images by separatingthe images using a parallax barrier, and others. It is possible that thestereoscopic image display determines the front and rear position (faror close) of the displayed object by the focal distance information(distance to the object) within the photographing time information.Moreover, as shown in FIG. 5 (FIG. 5 is a diagram corresponding to FIG.4(a)), when the object A is close to a observer E, the images of thecameras 2 and 5 are selected, and when the object A is far from theobserver E, the images of the cameras 3 and 4 are selected.

As described above, the present invention has an effect to rendervarious stereoscopic image displays possible.

1. The stereoscopic vision-use image providing method characterized inproviding, when providing a two-dimensional image as data, stereoscopicvision-use information useful for converting the data of saidtwo-dimensional image into a stereoscopic vision-use image and thicknessinformation of an object on said two-dimensional image, as additionalinformation of said two-dimensional image together with the data of saidtwo-dimensional image.
 2. The stereoscopic vision-use image providingmethod characterized in providing, when providing a two-dimensionalimage as data, stereoscopic vision-use information useful for convertingthe data of said two-dimensional image into a stereoscopic vision-useimage such as depth information indicating a near side position of anobject on said two-dimensional image and depth information indicating afar side position of the object on said two-dimensional image, asadditional information of said two-dimensional image together with thedata of said two-dimensional image.
 3. The stereoscopic vision-use imageproviding method characterized in providing, when providing atwo-dimensional image as data, stereoscopic vision-use informationuseful for converting the data of said two-dimensional image into astereoscopic vision-use image and thickness information of each dot onsaid two-dimensional image, as additional information of saidtwo-dimensional image together with the data of said two-dimensionalimage.
 4. The stereoscopic vision-use image providing methodcharacterized in providing, when providing a two-dimensional image asdata, stereoscopic vision-use information useful for converting the dataof said two-dimensional image into a stereoscopic vision-use image suchas depth information indicating a near side of each dot on saidtwo-dimensional image and depth information indicating a far side ofeach dot on said two-dimensional image, as additional information ofsaid two-dimensional data together with the data of said two-dimensionalimage.
 5. A stereoscopic vision-use image providing method according toany one of claims 1 to 4, characterized in providing information by anyone of methods such as broadcasting, a communication, and a recordinginto a recording medium.
 6. A stereoscopic vision-use image providingmethod according to any one of claims 1 to 5, characterized in providingat least one photographing time information out of focal distanceinformation and field angle information, as additional information ofsaid two-dimensional image together with the data of said twodimensional image.
 7. A stereoscopic vision-use image providing methodthat provides multi-viewpoint two-dimensional images as data,characterized in providing at least one photographing time informationout of information indicating the intervals between viewpoints,information indicating an angle formed of adjoining viewpoints and anobject to be photographed, information indicating a cross location ofoptical axes, focal distance information, and field angle information,as additional information of the two-dimensional image together with thedata of said two-dimensional image.
 8. A stereoscopic image displaydevice, comprising: a means for generating data of a stereoscopicvision-use image on the basis of data of a two-dimensional image andstereoscopic vision-use information; a means for composing an alternateimage with said stereoscopic vision-use image on the basis of data ofsaid alternate image; and a means for determining a collision between adisplayed object on the stereoscopic vision-use image and a displayedobject on said alternate image on the basis of thickness information ofdots and an object on said two-dimensional image that are additionalinformation of said two dimensional image.
 9. A stereoscopic imagedisplay device, comprising: a means for generating data of astereoscopic vision-use image on the basis of data of a two-dimensionalimage and depth information indicating a near side of an object on saidtwo-dimensional image; and a means for generating thickness informationof the object on the basis of depth information indicating a far sideposition of said object and said depth information indicating the nearside position of the object.
 10. A stereoscopic image display,comprising: a means for generating data of a stereoscopic vision-useimage on the basis of data of a two-dimensional image and depthinformation indicating a near side position of each dot on saidtwo-dimensional image; and a means for generating thickness informationof each dot on the basis of depth information indicating a far sideposition of said each dot and said depth information indicating the nearside position of said each dot.
 11. A stereoscopic image display thatperforms a stereoscopic image display using two images out ofmulti-viewpoint images, characterized in selecting said two images onthe basis of at least one photographing time information out ofinformation indicating intervals between viewpoints, informationindicating an angle formed of adjoining viewpoints and an object to bephotographed, information indicating a cross location of optical axes,focal distance information, and field angle information.